Tailored Mutants Fit Bugs Better

Mutation analysis measures test suite adequacy, the degree to which a test suite detects seeded faults (mutants): one test suite is better than another if it detects more mutants. The effectiveness of mutation analysis rests on the assumption that mutants are coupled with real faults—that is, mutant detection is strongly correlated with real fault detection. The work that validated this assumption also showed that a large portion of defects remain out of reach. We introduce tailored mutation operators to reach and capture these defects. Tailored mutation operators are built from and apply to an existing code base and its version history. They can, for instance, identify and replay errors specific to the project for which they are tailored. As our point of departure, we define tailored mutation operators for identifiers, which mutation analysis has largely ignored, because there are too many ways to mutate identifiers. Evaluated on the Defects4J data set, our new mutation operators allow mutation analysis to create mutants coupled to 14% more faults, compared to traditional mutation operators. These new mutation operators, however, quadruple the number of mutants, exacerbating the problem of mutant selection. To combat this problem, this paper proposes a new approach to mutant selection that focuses on the location at which to apply mutation operators and the unnaturalness of the mutated code. The results demonstrate that the location selection heuristics produce mutants more closely coupled to real faults for a given budget of mutation operator applications. In summary, this paper defines and explores tailored mutation operators, advancing the state of the art in mutation testing in two ways: 1) it suggests mutation operators that mutate identifiers and literals, extending mutation analysis to a new class of faults and 2) it demonstrates that selecting the location where a mutation operator is applied decreases the number of generated mutants without affecting the coupling of mutants and real faults. Keywords-mutation testing; submodular optimization; code naturalness